PCA Project¶
The purpose of the case study is to classify a given silhouette as one of four different types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars. The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Load libraries¶
In [1]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn import metrics
In [2]:
#Load the data
data_df = pd.read_csv('vehicle.csv')
In [3]:
data_df.head(10)
Out[3]:
First look at Data and Preprocessing¶
In [4]:
features = data_df.columns.tolist()[0:-1]
target = data_df.columns.tolist()[-1]
In [5]:
print("feature lists--> %s \n\n\ntarget--> %s"%(features,target))
In [6]:
data_df.shape
#we can see that there are 846 rows and 19 columns ( including a target column)
Out[6]:
In [7]:
data_df.info()
From the output we notice that there are missing values or NAN values for some of the features
In [8]:
#Lets check for NaN or null values
data_df.isna().any()
Out[8]:
In [9]:
#Lets count the NaNs
data_df.isna().sum(axis=0)
Out[9]:
In [10]:
#Lets fill NaNs with median
data_df.fillna(data_df.median(),inplace=True)
In [11]:
#Lets count the NaNs
data_df.isna().sum(axis=0)
#Great, all null values are taken care now
Out[11]:
In [12]:
data_df.describe().transpose()
Out[12]:
In [13]:
data_df['class'].value_counts()
Out[13]:
From the output above we can infer
- scatter_ratio is slighty right skewed as mean value is > median, same case with scaled_variance, scaled_variance.1
- rest all features looks to be normally distributed
In [14]:
# Lets check outliers using boxplot
data_df.boxplot(figsize=(20,5),rot=50)
Out[14]:
From the graph we can see that
- radius_ration , pr.axis_aspect_ratio , max.length_aspect_ration ,scaled_variance, scaled_variance.1 , scaled_radius_of_gyration features have outliers
In [15]:
# Lets take care of outliers now
# as per 5 number summary , we know that the values within 3 std from mean account for about 99.7% of the data set
# lets use this aproach to knock out data that fall below or above 3 std cutoff ( we can replace outliers with median values too)
In [16]:
# make copy of original data set
newdata = data_df.copy()
In [17]:
newdata.head(3)
Out[17]:
In [18]:
# Taking care of radius_ratio outliers
data_mean = newdata['radius_ratio'].mean()
cutoff = newdata['radius_ratio'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['radius_ratio'] > upper) | (newdata['radius_ratio'] < lower)) ]
#So these are the outliers for radius_ration which we have to remove
Out[18]:
In [19]:
#Remove the outliers indexes
newdata.drop(newdata[((newdata['radius_ratio'] > upper) | (newdata['radius_ratio'] < lower)) ].index,inplace=True)
In [20]:
# Taking care of pr.axis_aspect_ratio outliers
data_mean = newdata['pr.axis_aspect_ratio'].mean()
cutoff = newdata['pr.axis_aspect_ratio'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['pr.axis_aspect_ratio'] > upper) | (newdata['pr.axis_aspect_ratio'] < lower)) ]
#So these are the outliers for pr.axis_aspect_ratio which we have to remove
Out[20]:
In [21]:
#Remove the outliers indexes
newdata.drop(newdata[((newdata['pr.axis_aspect_ratio'] > upper) | (newdata['pr.axis_aspect_ratio'] < lower)) ].index,inplace=True)
In [22]:
# Taking care of max.length_aspect_ratio outliers
data_mean = newdata['max.length_aspect_ratio'].mean()
cutoff = newdata['max.length_aspect_ratio'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['max.length_aspect_ratio'] > upper) | (newdata['max.length_aspect_ratio'] < lower)) ]
#So these are the outliers for pr.axis_aspect_ratio which we have to remove
Out[22]:
In [23]:
#Remove the outliers indexes
newdata.drop(newdata[((newdata['max.length_aspect_ratio'] > upper) | (newdata['max.length_aspect_ratio'] < lower)) ].index,inplace=True)
In [24]:
# Taking care of scaled_variance outliers
data_mean = newdata['scaled_variance'].mean()
cutoff = newdata['scaled_variance'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['scaled_variance'] > upper) | (newdata['scaled_variance'] < lower)) ]
#So these are the outliers for scaled_variance which we have to remove
Out[24]:
In [25]:
#Remove the outliers indexes
newdata.drop(newdata[((newdata['scaled_variance'] > upper) | (newdata['scaled_variance'] < lower)) ].index,inplace=True)
In [26]:
# Taking care of scaled_variance.1 outliers
data_mean = newdata['scaled_variance.1'].mean()
cutoff = newdata['scaled_variance.1'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['scaled_variance.1'] > upper) | (newdata['scaled_variance.1'] < lower)) ]
#So these are the outliers for scaled_variance.1 which we have to remove
Out[26]:
In [27]:
#Remove the outliers indexes
newdata.drop(newdata[((newdata['scaled_variance.1'] > upper) | (newdata['scaled_variance.1'] < lower)) ].index,inplace=True)
In [28]:
# Taking care of scaled_radius_of_gyration outliers
data_mean = newdata['scaled_radius_of_gyration'].mean()
cutoff = newdata['scaled_radius_of_gyration'].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata['scaled_radius_of_gyration'] > upper) | (newdata['scaled_radius_of_gyration'] < lower)) ]
#So these are the outliers for scaled_radius_of_gyration which we have to remove, nothing to remove
Out[28]:
In [29]:
#We can also use impute method to replace outliers with median, below method can be used for the same
def outlierTreatment(feature):
# Taking care of outliers
data_mean = newdata[feature].mean()
cutoff = newdata[feature].std()*3
lower ,upper = data_mean - cutoff , data_mean + cutoff
lower,upper
newdata[((newdata[feature] > upper) | (newdata[feature] < lower)) ]
#So these are the outliers which we have to remove
#Remove the outliers indexes
#newdata.drop(newdata[((newdata[feature] > upper) | (newdata[feature] < lower)) ].index,inplace=True)
#update outliers with median
newdata.loc[((newdata[feature] > upper) | (newdata[feature] < lower)),feature ] = newdata[feature].median()
All the outliers are now taken care, lets plot box plot again to see¶
In [30]:
newdata.boxplot(figsize=(20,8),rot=50)
Out[30]:
Understanding the attributes, univariate and multivariate analysis¶
Lets plot histogram to understand the distribution of dataset¶
In [31]:
count = 1
plt.figure(figsize=(15,15))
for feature in features:
plt.subplot(6,3,count)
plt.tight_layout()
plt.ylabel('Fequency Distribution')
sns.distplot(newdata[feature])
count = count +1
plt.plot()
Out[31]:
From the graph above
- we can see that compactness and scaled_radius_of_gyration looks normally distirbuted
- some of the feature are skewed slighly like pr.axis_aspect_ratio, skewness_about.1
- we also see that many features have multiple density curves(binomial distribution) , circularity,distance_circularity,scatter_ratio,scaked+variance etc
- we also see random distribution in some case like hollows_ratio
Lets now plot pairplot to understand if features are correlated¶
In [32]:
sns.pairplot(newdata[features])
Out[32]:
From the above pair plot we can see that¶
- there is positive correlation between compactness and circularity
- There are many features which are correlated , either have postive or negative correlation
- There are few features like hollows_ratio,skewness_about.1 etc have no correlation as we see cloud like scatter plot distributions
In [33]:
#Lets check with heat map
plt.figure(figsize=(20,15))
sns.heatmap(newdata[features].corr(),annot=True)
Out[33]:
In [34]:
#Lets check correlation values, value close to 0 no correlation value close to (+/-) 1 has correlation
newdata[features].corr()
Out[34]:
some of the facts we know¶
- we know that, model will perform well if all the features are independent or have no correlation, however with current dataset this is not the case as we have many features which are correlated
- PCA ( Principal Component Analysis) is one approach or technique that is used to increase the signle to noise ration and for dimension reduction, this technique can be used to get the information content that is available in mathematical space but never fed to the model ( infact this technique works well on features having correlation by extracting the information that is no where fed to model or used to build the model)
Let us now use the PCA to select the features (though it selects the features internally) to build the Principal components which are then used to build the model, below are the steps that needs to be done¶
- Standardize the data
- Generate the covariance matrix , value close to 0 no correlation value close to (+/-) 1 has correlation
- Perform eigen decomposition, compute eigen vectors which are the principal components and the corresponding eigen values which are the magnitudes of variance captured
- Sort the eigen pairs in decending order and select the largest one
- Eigne vectors with insignificant contribution to total eigen values can be removed from analysis
Standardize the Data , use zscore¶
In [35]:
from scipy.stats import zscore
data_std = newdata[features].apply(lambda a:zscore(a))
In [36]:
data_std.head()
Out[36]:
Lets genereate covariance matrix/Correlation matrix¶
In [37]:
#Lets check correlation values, value close to 0 no correlation value close to (+/-) 1 has correlation
#data_std.corr()
#we can use below as well
cov_matrix = np.cov(data_std.T)
print('Covariance Matrix \n%s', cov_matrix)
Perform Eigen decomposition¶
In [38]:
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s' %(eigenvectors))
print('\n Eigen Values \n%s' %(eigenvalues))
- At this point eigen vectors/Principal components are formed ( eigen vectors are nothing but Principal Components)
- number of eigen vectors generated will be equal to the number of indepdnent features
- every eigen vector is associated with eigen values
Lets sort eigen vectors / PCAs¶
In [39]:
#Lets plot cumulative explained variance
from sklearn.decomposition import PCA
pca = PCA().fit(data_std) # this actually generates eigen vectors(Principal components) and eigen values
plt.plot(np.cumsum(pca.explained_variance_ratio_))
Out[39]:
In [40]:
#Lets print the same
np.cumsum(pca.explained_variance_ratio_)
Out[40]:
- from the output above we can read as PC1 will capture around 55% of information from original mathematical space
- like wise PC1 and PC2 together capture around 73% of information
- PC1 to PC12 all together capture around 99.4 % of the information
- So lets use 12 PCs , that is reducing to 12 dimension ( originally we had 18 dimensions)
Eigne vectors with insignificant contribution to total eigen values can be removed from analysis¶
In [41]:
#Lets use 12 dimensions to build PCs , as these capture around 99.4 % of the information as we saw previously
pca = PCA(n_components=12)
pca.fit(data_std)
data_std_pca = pca.transform(data_std)
In [42]:
data_std_pca = pd.DataFrame(data_std_pca)
data_std_pca.head()
# so below is the transformed data using 12 principal components
Out[42]:
In [43]:
# Lets pair plot to see the output now
sns.pairplot(data_std_pca,diag_kind='kde')
Out[43]:
- from the output , we can see that all the data looks like clould, which means between PCs there is no relation ship
- so PCA has done good work here , it has removed all the covariance or very close to zero
Lets now use these Principal Components to build the model¶
Train Test split of PCA transformed data¶
In [44]:
from sklearn.model_selection import train_test_split
In [45]:
x_train,x_test,y_train,y_test = train_test_split(data_std_pca, newdata[target],test_size=0.3,random_state=28)
In [46]:
print(x_train.shape,x_test.shape)
print(y_train.shape,y_test.shape)
In [47]:
def evaluateModel(model,name):
print("######Evaluate the model %s ###########"%(name))
print("Training Accuracy score : \t %s "%(model.score(x_train,y_train)))
print("Test Accuracy score : \t %s "%(model.score(x_test,y_test)))
print("Confusion matrix \n\t %s"%(metrics.confusion_matrix(y_test,y_pred)))
print("classicification report \n %s"%(metrics.classification_report(y_test,y_pred)))
Lets use Naive Bayes algorithm to build the model on PCA transformed data¶
In [48]:
model_nb = GaussianNB()
model_nb.fit(x_train,y_train)
y_pred = model_nb.predict(x_test)
model_nb.score(x_test,y_test)
Out[48]:
In [49]:
#Lets evaluate the model
evaluateModel(model_nb, "Naive Bayes")
we can see that train accuracy is 89% and test accuracy is 87% [ PCA Transformed Data ]¶
Lets use Naive Bayes algorithm to build the model on Original data (standardized but not PCA transformed)¶
Train Test split of original data (standardized) data_std¶
In [55]:
x_train,x_test,y_train,y_test = train_test_split(data_std, newdata[target],test_size=0.3,random_state=28)
In [56]:
model_nb = GaussianNB()
model_nb.fit(x_train,y_train)
y_pred = model_nb.predict(x_test)
model_nb.score(x_test,y_test)
Out[56]:
In [57]:
#Lets evaluate the model
evaluateModel(model_nb, "Naive Bayes")
we can see that train accuracy is 64% and test accuracy is 62% [ Non PCA Transformed data]¶
Lets use Support Vector Classifier algorithm to build the model on PCA transformed data¶
Train Test split of PCA transformed data¶
In [58]:
x_train,x_test,y_train,y_test = train_test_split(data_std_pca, newdata[target],test_size=0.3,random_state=28)
In [59]:
model_svm = SVC(kernel='rbf')
model_svm.fit(x_train,y_train)
y_pred = model_svm.predict(x_test)
model_svm.score(x_test,y_test)
Out[59]:
In [51]:
#Lets evaluate the model
evaluateModel(model_svm, "Support vectore classifier")
we can see that train accuracy is 98% and test accuracy is 95% [ PCA Transformed data ]¶
Lets use Support Vector Classifier algorithm to build the model on original data standardized but not PCA transformed¶
Train Test split of original data (standardized) data_std¶
In [60]:
x_train,x_test,y_train,y_test = train_test_split(data_std, newdata[target],test_size=0.3,random_state=28)
In [61]:
model_svm = SVC(kernel='rbf')
model_svm.fit(x_train,y_train)
y_pred = model_svm.predict(x_test)
model_svm.score(x_test,y_test)
Out[61]:
In [62]:
#Lets evaluate the model
evaluateModel(model_svm, "Support vectore classifier")
we can see that train accuracy is 97% and test accuracy is 95% [ Non PCA Transformed data ]¶
#
Summary¶
- Data Preprocessing
- As part of data preprocessing we took a glance over number of independent features and number of overall records(rows) available as part of the dataset
- we found out that some of the independent features had missing values, so in order to fix these we replaced all of the missing values with median of that corresponding feature
- we also checked the statistical information of features by using describe() method and noticed that only few features like scatter_ratio, scaled_variance were skewed in nature and rest features looked as normally distributed
- we then checked for outliers using boxplot, we found out that some of the features had outliers and we used 5 number summary to identify the data set that needs to be considered as outliers and then removed them from the original data set ( we can also replace these data sets with median value of the feature)
- UnderStanding the attributes/features
- we draw histogram plot of each features to understand the distributions of data, found out that distribution was mixed in nature, few feature sets were normally distributed, some were binomial and random distribution in nature (having two and more kde density curve)
- then we used pair plot to understand the relationship between independent variables, found out many features were correlated either positively or negatively
- To cross check we used heatmap to see the correlation
- Principal component analysis
- Ideally features which have correlation are considered in PCA, however in this usecase we have used entire set of indepenent features ( though we can actually select only those feature which have correlation from pairplot observations)
- Since PCA helps to increase signle to noise ration ( improves the information content by capturing the info that is availalbe in mathematical space but not used) and also used to reduce the dimension, we considered this for our analysis
- to start with we standardized the complete set of independent features using zscore
- Then generated covariance/correlation matrix , and also generated Principal compoments (eigen vectors) & eigen values
- we then plot cumulative explained variance of Principal components to understand how each PCs contribute to reduce the variance
- from the graph and explained variance values we found out that 12 principal components together contribute 99.4% in reducing the variance
- we then used 12 Principal component to transform the dataset and stored it in new dataframe
- we used pairplot to check relationship of these principal components and noticed that there was not relation ( so PCA has done good work)
Model building
- To build the model we choose Naive Bayes and Support Vector Classifier algorithm
- ### Model building using Naive Bayes algorithm
Without Principal components( original dataframe, standardized) , Naive Bayes algoritm gave us overall test score of 62%¶
With Principal components ( transformed data), Naive Bayes algorithm gave us good result with overall test score of 87%¶
- ### Model building using Support Vector Classifier algorithm
Without Principal components( original dataframe, standardized) , Support Vector Classifier algorithm gave us overall test score of 95.9%¶
With Principal components ( transformed data), Support Vector Classifier algorithm gave us almost same result with overall test score of 95.5%¶
Final statement
Principal components ( PCA ) in general will give best results when used in model than the original data set, as PCA makes sure to increase the information content by fetching the information from mathematical space that is not used or fed to the model As PCA works best on features having positive/negative correlation , Principal components may not yeild good results if these are not taken care in PCA ( choose only feature having relationship between them to generate PCs) In our usecase we took all the features for PCA , and noticed that Naive Bayes algorithm preformed well on Principal components